# Self-Supervised Learning

SHMT is a self-supervised hierarchical makeup transfer technology achieved through latent diffusion models. This technology allows for the natural transfer of one facial makeup to another without the need for explicit labeling. Its main advantages include the ability to handle complex facial features and expression changes, providing high-quality transfer results. This technology has been accepted at NeurIPS 2024, showcasing its innovation and practicality in the field of image processing.

1.58-bit FLUX is an advanced text-to-image generation model that employs 1.58-bit weights (values of {-1, 0, +1}) to quantify the FLUX.1-dev model while maintaining comparable performance for generating 1024x1024 images. This method does not require access to image data and relies entirely on the self-supervision of the FLUX.1-dev model. Additionally, a custom kernel has been developed to optimize 1.58-bit operations, achieving a 7.7x reduction in model storage, a 5.1x decrease in inference memory, and improved inference latency. Extensive evaluations in GenEval and T2I Compbench benchmarks show that 1.58-bit FLUX significantly enhances computational efficiency while maintaining generation quality.

Video-Foley is an innovative system for generating sound from video. It employs root mean square (RMS) as a temporal event condition, combined with semantic tonal prompts (audio or text), to achieve high control and synchronization in video sound synthesis. The system utilizes an unsupervised learning framework that requires no manual labeling, consisting of two stages: Video2RMS and RMS2Sound, incorporating novel concepts such as RMS discretization and RMS-ControlNet, in conjunction with a pre-trained text-to-audio model. Video-Foley achieves state-of-the-art performance in aligning and controlling sound timing, intensity, timbre, and detail.

HOI-Swap is a diffusion model-based video editing framework specializing in tackling the complexities of hand-object interactions in video editing. This model, trained through self-supervision, enables seamless object swapping within a single frame. It also learns to adjust hand interaction patterns based on object attribute changes, such as grip style. The second stage extends single-frame editing to an entire video sequence, achieving high-quality video editing through motion alignment and video generation.

MimicBrush is an innovative image editing model that allows users to achieve zero-shot image editing by specifying the editing area in the source image and providing a reference image. The model can automatically capture the semantic correspondence between the two images and complete the editing in one step. MimicBrush is developed based on diffusion priors, capturing semantic relationships between different images through self-supervised learning, and experimental results demonstrate its effectiveness and superiority in various test cases.

DenseAV is a novel dual-encoder localization architecture that learns high-resolution, semantically meaningful audio-visual alignment features by observing videos. It can discover the "meaning" of words and the "location" of sounds without requiring explicit localization supervision, and automatically discovers and distinguishes between these two types of associations. DenseAV's localization capability stems from a new multi-head feature aggregation operator, which directly compares dense image and audio representations through contrastive learning. Additionally, DenseAV significantly outperforms previous art on semantic segmentation tasks and surpasses ImageBind in cross-modal retrieval using less than half the parameters.

AniTalker is an innovative framework that can generate realistic dialogue facial animations from a single portrait. It enhances expressive motion capture through two self-supervised learning strategies, and develops an identity encoder through metric learning, effectively reducing the need for labeled data. AniTalker not only creates detailed and realistic facial expressions but also emphasizes its potential in real-world applications for producing dynamic avatars.

Miqu 1-70b is an open-source large language model that utilizes novel self-supervised learning methods, capable of handling a variety of natural language tasks. With 170 billion parameters, it supports multiple prompt formats and allows for fine-tuning to generate high-quality text. Its strong comprehension and generation capabilities make it suitable for applications in chatbots, text summarization, question answering systems, and other domains.

This paper systematically evaluates the ability of large language models (LLMs) to generate and recognize increasingly complex visual concepts, and demonstrates how to train initial visual representation learning systems using text models. Although language models cannot directly process pixel-level visual information, this research utilizes code representations of images. While LLM-generated images are not like natural images, the results on image generation and correction suggest that accurately modeling strings can teach language models much about the visual world. Furthermore, experiments on self-supervised visual representation learning using text-model generated images highlight the potential of training visual models capable of semantic evaluation on natural images using only LLMs.